Skip to content

test(e2e): instrument and diagnose bot suite setup cost#24534

Merged
PhilWindle merged 1 commit into
merge-train/spartan-v5from
spl/e2e-bot-setup-spans
Jul 8, 2026
Merged

test(e2e): instrument and diagnose bot suite setup cost#24534
PhilWindle merged 1 commit into
merge-train/spartan-v5from
spl/e2e-bot-setup-spans

Conversation

@spalladino

@spalladino spalladino commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

What / why

Round-4 e2e speedup, PR 3 of the effort. The single-node/bot suite's file-level beforeAll costs
~178s on CI with 97% untagged, and the residual proven-chain waits in the private_payments/failures
fees suites (~32s/shard) were also invisible. This PR instruments both so the cost is attributable, and
diagnoses the bot residual.

Spans are pure passthrough when TEST_TIMING_FILE is unset (see fixtures/timing.ts), so this is zero
behavior change for normal runs and CI.

Commit

  • test(e2e): instrument bot suite setup and fees proven-chain residuals
    • Bot suite (single-node/bot/bot.test.ts): setup:wallet around EmbeddedWallet.create,
      wallet:create around createSchnorrInitializerlessAccount, and setup:bot around the four nested
      describe beforeAll hooks (Bot.create / AmmBot.create / CrossChainBot.create / BotStore).
    • Fees harness (single-node/fees): wait:proven-checkpoint around catchUpProvenChain's poll loop,
      and warp:proven-checkpoint-epoch around advanceToNextEpoch via a new instrumented
      FeesTest.advanceToNextEpoch() wrapper; waitForEpochProven and the direct call sites in
      failures/private_payments now route through it.
    • Reuses existing tags where the concept matches (wallet:create is already what
      createFundedInitializerlessAccounts fires; warp:proven-checkpoint-epoch is already what
      advanceToNextEpoch fires in block-production/setup.ts) rather than minting near-duplicates.

Diagnosis

The timing environment (shared/timing_env.mjs) folds every beforeAll in a file — the file-level
one plus each nested describe's — into a single suite-scoped beforeHooksMs, and every span fired
during any beforeAll lands on that same suite line. The shortlist read the whole number as the
file-level hook and, seeing only setup:env:* tagged, attributed the rest to the one visible untagged
file-level call (createSchnorrInitializerlessAccount). It is actually the nested hooks.

Local read (TEST_TIMING_SPANS=1, PIPELINING_SETUP_OPTS = 12s L2 slots, production sequencer, fake
prover), suite beforeHooksMs = 153,733 ms:

  • setup:env:* (file-level setup()): ~4.6s — already tagged
  • setup:wallet (EmbeddedWallet.create, ephemeral): 137 ms — the PXE has autoSync:false, no
    genesis walk
  • wallet:create (createSchnorrInitializerlessAccount): 69 ms — initializerless, no deploy tx, as
    designed
  • setup:bot (4 nested hooks): 149,582 ms = 97.3% of the file's beforeAll
    • transaction-bot Bot.create: 32.2s (token deploy class+instance, then private+public mint)
    • bridge-resume new BotStore(...): 0.9 ms
    • amm-bot AmmBot.create: 73.9s (deploys 4 contracts + set_minter + mint + add_liquidity)
    • cross-chain-bot CrossChainBot.create: 43.5s (TestContract deploy + seed 2 L1→L2 msgs + wait ready)

Root cause: structural, not a bug. The bot suite runs the production Sequencer to exercise
CHECKPOINTED/PROPOSED follow modes and L1 fee-juice bridging, so each Bot.create deploys real
contracts and mints serially at slot cadence (bot/src/factory.ts). The two calls the shortlist
suspected are ~0.1s each.

No fix is shipped here because there is no safe one-liner: speeding this up means batching/parallelizing
the bot/src/factory.ts deploy/mint paths (production code, PR-6-shaped), genesis-seeding (PR 1/2), or
cutting the CI slot time (PR 5) — none belong in the instrumentation PR. Tracked as a round-5 item; the
new setup:bot span makes that win measurable. Full write-up in tmp/SPEEDUP_ROUND4_FINDINGS.md.

Expected effect

No wall-clock change (passthrough spans). The deliverable is visibility: the previously-invisible
setup:bot / setup:wallet / wallet:create decomposition of the bot beforeAll, and the
wait:proven-checkpoint / warp:proven-checkpoint-epoch residual in the fees suites.

Measured numbers from a full green CI run will be appended below once available.

Measured impact

Both runs are full CI runs. PR run: ci/x-fast on head 1f7898cd2b (CI 1783345614459429, 1,799 test
lines). Baseline: ci/x-full-no-test-cache on the exact merge-base a8cfa93df5 (CI 1783341726388943,
1,863 test lines). No wall-clock change claimed (spans are passthrough; suite-level noise floor is
~11s median / ~52s p90) — the deliverable is attribution, measured as span busyMs.

Bot suite (suite line, beforeAll = 181.5s on the PR run vs 182.1s baseline):

  • Baseline: 7.4s tagged (setup:env:* only) → 174.8s / 96% invisible.
  • PR run: setup:bot 174,203 ms (4 occurrences, max 86,961 ms = the amm-bot hook),
    setup:wallet 250 ms, wallet:create 144 ms; residual now −1.5s ≈ 0
    100% of the beforeAll is attributed.
  • This confirms the diagnosis: EmbeddedWallet.create + createSchnorrInitializerlessAccount are
    ~0.4s combined; the "invisible 178s" is the four nested describes' bot-factory setup
    (deploys + mints at production 12s-slot cadence), folded into the file's suite line by the
    hook accounting.

Fees suites (private_payments.parallel × 8 shards + failures):

  • Baseline: avg 32.3s/shard untagged residual (zero wait:proven-checkpoint /
    warp:proven-checkpoint-epoch lines on these suites).
  • PR run: wait:proven-checkpoint fires on all 9 suite hooks, 293,888 ms total
    (avg 32.7s/shard — matches the residual exactly), plus one in-test occurrence (32.1s);
    warp:proven-checkpoint-epoch 19 occurrences, 361 ms total (the warp RPC itself is
    near-free — the cost is entirely the proven-chain poll that follows). Per-shard residual is now
    ≈ −1s, i.e. fully attributed.

Net: ~469s/run of previously invisible setup/wait cost is now tagged and attributable
(174.2s setup:bot + 0.4s wallet spans + 294.3s fees proven-chain waits), with no behavior or
wall-clock change.

Fixes A-1407

@spalladino spalladino added the ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure label Jul 6, 2026
Adds timing spans so the untagged residual of the bot suite's file-level
beforeAll (~178s on CI, 97% invisible) can be decomposed and the ~32s/shard
proven-chain residual in the private_payments/failures fees suites becomes
visible.

Bot suite (single-node/bot/bot.test.ts):
- setup:wallet around EmbeddedWallet.create (stands up the ephemeral PXE)
- wallet:create around createSchnorrInitializerlessAccount (same concept as
  createFundedInitializerlessAccounts, already tagged wallet:create)
- setup:bot around the nested describes' Bot.create/AmmBot.create/
  CrossChainBot.create/BotStore creation, which the suite-line accounting folds
  into the same suite beforeHooksMs as the file hook

Fees harness (single-node/fees):
- wait:proven-checkpoint around catchUpProvenChain's poll loop
- warp:proven-checkpoint-epoch around advanceToNextEpoch, via a new instrumented
  FeesTest.advanceToNextEpoch() wrapper that waitForEpochProven and the direct
  call sites in failures/private_payments now route through

Reuses existing tags where the concept matches rather than inventing
near-duplicates. Spans are passthrough when TEST_TIMING_FILE is unset, so this
is zero behavior change.
@spalladino
spalladino force-pushed the spl/e2e-bot-setup-spans branch from 1f7898c to 46074c2 Compare July 6, 2026 21:50
@spalladino spalladino added wip Work in progress and removed wip Work in progress labels Jul 7, 2026
@PhilWindle
PhilWindle merged commit 1476179 into merge-train/spartan-v5 Jul 8, 2026
32 checks passed
@PhilWindle
PhilWindle deleted the spl/e2e-bot-setup-spans branch July 8, 2026 09:59
PhilWindle pushed a commit that referenced this pull request Jul 8, 2026
Parallelizes the independent setup steps in the bot factory
(`@aztec/bot`, `bot/src/factory.ts`).
This is production bot code that runs against live networks, so the
changes are strictly
behavior-preserving: same contracts at the same (salt-derived)
addresses, same amounts, same final
state. Only the ordering/concurrency of provably-independent setup steps
changes.

The bot factory is the single largest cost in the `single-node/bot` e2e
suite: each `*.create` runs
the production sequencer at 12s L2 slots (`PIPELINING_SETUP_OPTS`) and
deploys/mints one tx per slot,
fully serial. `AmmBot.create` alone was ~74s (7 serial txs).

## Dependency graph per bot type

Edges below are "must be mined before", verified against the Noir
contract source
(`token_contract`, `amm_contract`).

### Transaction bot -- `setup()` (unchanged)

`setupAccount` -> register recipient -> `ensureFeeJuiceBalance` ->
deploy token -> mint. This chain is
fully data-dependent: funding gates the deploy, the deploy gates the
mint. The recipient
(`createSchnorrAccount`) is register-only (no tx), and the
private+public mints are already batched
into one tx. Nothing is independent, so `setup()` is left serial.

### AMM bot -- `setupAmm()`

Steps: deploy token0 (D0), token1 (D1), LP token (DL), AMM (DA);
`set_minter` granting the AMM rights
over the LP token (SM); mint token0+token1 to the provider (M, one
batched tx); `add_liquidity` (AL).

- D0, D1, DL: independent -- distinct contracts, no cross-references.
- DA: the AMM constructor only stores the three token addresses (no
calls into the tokens). Those
addresses are salt-derived, so DA needs only the (pre-derived)
addresses, not the token deploys mined.
- SM: `Token::set_minter` just writes `minters[amm] = true`; it does not
call or validate the AMM
contract. It needs the LP token deployed and the (derivable) AMM address
only -- not the AMM deployed.
The deployer is authorized because the Token constructor sets
`minters[admin] = true`.
- M: `mint_to_private`/`mint_to_public` require the caller be a minter;
the deployer is admin=minter
from the constructor, so the token0/token1 mints need only those tokens
deployed (no `set_minter`).
- AL: `add_liquidity` transfers token0/token1 from the provider (needs M
for balances), mints LP
tokens (needs SM so the AMM is an LP minter), and targets the AMM (needs
DA).

Restructured from 7 serial txs into 3 slot-phases:

- Phase 1: `Promise.all([D0, D1, DL])`
- Phase 2: `Promise.all([DA, SM, M])`
- Phase 3: `AL`

### Cross-chain bot -- `setupCrossChain()`

Steps: deploy TestContract (an L2 tx), seed N L1->L2 messages (L1 txs),
wait for the first message ready.

- The seeds reference only the L2 recipient (TestContract) address,
which is salt-derived. L1->L2
messages are queued on L1 and do not require the L2 contract to exist
yet; they are consumed later in
`bot.run()`, after setup completes. So the L2 deploy and the L1 seeding
are independent.
- The L2 deploy pays via the L2 wallet/PXE account; the seeds use the
bot's L1 client -- different
  accounts, so there is no L1 nonce interaction between them.

Restructured to overlap the deploy with the seed loop:
`Promise.all([deploy TestContract, seed loop])`,
then the message-ready wait.

## What stayed serial, and why

- The whole transaction-bot `setup()` (data-dependent chain, nothing
independent).
- `ensureFeeJuiceBalance` before every deploy (funding must precede
spending).
- `add_liquidity` after its mints + minter grant + AMM deploy.
- The individual L1->L2 seeds inside the loop: they share the bot's
single L1 account, so concurrent
sends would race on the L1 nonce. The suite already documents this nonce
hazard. Only the loop as a
  whole overlaps the (L2) TestContract deploy.

## Production safety / idempotent re-entry

- Every deploy still goes through `registerOrDeployContract`, which
checks
`getContractMetadata(address).isContractPublished` per contract and only
registers (no tx) if already
deployed. If one parallel deploy fails and the others succeed, the next
`*.create` recovers: the
succeeded contracts register-only, the failed one redeploys. Addresses
are salt-derived and identical
  across runs.
- `set_minter` is idempotent (writes `true` again). The AMM path's mint
and `add_liquidity` always run
  (no balance guard) -- unchanged from the previous `fundAmm`.
- Same-sender concurrent `.send()`s are safe by construction: the PXE
serializes simulate/prove through
its `SerialQueue` and setup txs pay with random tx nonces. The bot runs
the same `EmbeddedWallet` ->
  PXE path in the e2e suite and in production.
- Deliberate, benign failure-path change: because Phase 2 runs DA/SM/M
concurrently, a failure of one no
longer prevents the others from being sent. Their effects (a
deployed-but-unused AMM, a minter grant,
an extra mint) are all idempotent-safe, and `add_liquidity` uses fixed
amounts, so re-entry converges
  to the same final state.
- `withNoMinTxsPerBlock` (the `flushSetupTransactions` save/zero/restore
wrapper around each setup tx)
was not safe to re-enter concurrently: interleaved save/zero/restore
could read the already-zeroed
value and "restore" `minTxsPerBlock` to 0 permanently. It now
reference-counts entrants -- the first
saves and zeroes, the last restores -- with unit tests covering the
overlapping and failure paths
(`bot/src/factory.test.ts`). Serial callers see the exact same RPC
sequence as before.

Final state is identical in every path: same contract addresses, same
minter grant, same minted
amounts, same liquidity.

## Local evidence

Two full local runs of `single-node/bot/bot.test.ts` (production
sequencer, 12s L2 slots), all 10
tests passing in both. Hook durations from factory log timestamps,
against the round-4 local baseline
measured on the same machine at the same cadence (findings from #24534):

- `Bot.create` (transaction-bot hook, untouched): 31.8s vs 32.2s
baseline -- unchanged, as intended.
- `AmmBot.create`: 57.2s vs 73.9s baseline (-16.7s). The three token
deploy txs go out within 300ms of
each other; the AMM deploy, `set_minter` and the mint batch all go out
within the following slot.
- `CrossChainBot.create`: 24.0s vs 43.5s baseline (-19.5s). The
TestContract deploy overlaps both L1
seeds; the first message is ready almost immediately after the deploy is
mined.
- Suite-level `beforeHooksMs`: 116.9s vs 153.7s baseline (-36.8s),
matching the per-hook deltas.
- Unit: `bot/src/factory.test.ts` (4 tests) covering the
`withNoMinTxsPerBlock` reentrancy fix.

## Expected effect on CI

- `AmmBot.create`: 7 serial txs -> 3 slot-phases (~87s on CI per
#24534's spans -> ~55-60s).
- `CrossChainBot.create`: deploy overlaps the seed pipeline (~43s ->
~25s).
- Transaction bot: unchanged.
- Aggregate: roughly -35 to -45s on the bot suite's beforeAll wall time.
- Fix-phase proof metric (once #24534's `setup:bot` instrumentation is
on this base): the amm-bot and
cross-chain `setup:bot` occurrences drop per the numbers above. Measured
CI numbers to be appended
  when a full run lands.


## Measured impact

The bot suite (a single shard) drops 216.3s → 118.3s total (**−98.0s,
−45%**); beforeHooks 215.9s →
118.0s.

- No `setup:bot` spans exist on this base (that instrumentation lives in
the unmerged #24534), so the
metric is the suite-level hook fold, which covers all three nested bot
factory setups (Bot, AmmBot,
  CrossChainBot).
- Noise context from the same run pair: untouched light suites move −2
to −3s per shard
(private_initialization −2.9s/shard over 20 shards, deploy_method
−2.1s/shard over 14), and the
largest counter-move is validators_sentinel +24s (multi-node variance).
A −98s single-shard delta is
  far outside that envelope.
- The CI delta exceeds the local −36.8s because the CI baseline pays
more missed-slot penalties per
serial tx (baseline beforeHooks 215.9s on CI vs 153.7s locally at the
same 12s cadence); collapsing
7 serial txs into 3 slot-phases removes proportionally more where slots
are more often missed.

Baseline CI run 1783393028773172 (merge-base 30966a4, full). PR CI run
1783427250198215 (x-fast).


Fixes A-1408
PhilWindle pushed a commit that referenced this pull request Jul 21, 2026
## What / why

Round-4 e2e speedup, PR 3 of the effort. The `single-node/bot` suite's
file-level `beforeAll` costs
~178s on CI with 97% untagged, and the residual proven-chain waits in
the `private_payments`/`failures`
fees suites (~32s/shard) were also invisible. This PR instruments both
so the cost is attributable, and
diagnoses the bot residual.

Spans are pure passthrough when `TEST_TIMING_FILE` is unset (see
`fixtures/timing.ts`), so this is zero
behavior change for normal runs and CI.

## Commit

- `test(e2e): instrument bot suite setup and fees proven-chain
residuals`
- Bot suite (`single-node/bot/bot.test.ts`): `setup:wallet` around
`EmbeddedWallet.create`,
`wallet:create` around `createSchnorrInitializerlessAccount`, and
`setup:bot` around the four nested
`describe` beforeAll hooks (`Bot.create` / `AmmBot.create` /
`CrossChainBot.create` / `BotStore`).
- Fees harness (`single-node/fees`): `wait:proven-checkpoint` around
`catchUpProvenChain`'s poll loop,
and `warp:proven-checkpoint-epoch` around `advanceToNextEpoch` via a new
instrumented
`FeesTest.advanceToNextEpoch()` wrapper; `waitForEpochProven` and the
direct call sites in
    `failures`/`private_payments` now route through it.
- Reuses existing tags where the concept matches (`wallet:create` is
already what
`createFundedInitializerlessAccounts` fires;
`warp:proven-checkpoint-epoch` is already what
`advanceToNextEpoch` fires in `block-production/setup.ts`) rather than
minting near-duplicates.

## Diagnosis

The timing environment (`shared/timing_env.mjs`) folds *every*
`beforeAll` in a file — the file-level
one plus each nested `describe`'s — into a single suite-scoped
`beforeHooksMs`, and every span fired
during any beforeAll lands on that same suite line. The shortlist read
the whole number as the
file-level hook and, seeing only `setup:env:*` tagged, attributed the
rest to the one visible untagged
file-level call (`createSchnorrInitializerlessAccount`). It is actually
the nested hooks.

Local read (`TEST_TIMING_SPANS=1`, `PIPELINING_SETUP_OPTS` = 12s L2
slots, production sequencer, fake
prover), suite `beforeHooksMs = 153,733 ms`:

- `setup:env:*` (file-level `setup()`): ~4.6s — already tagged
- `setup:wallet` (`EmbeddedWallet.create`, ephemeral): **137 ms** — the
PXE has `autoSync:false`, no
  genesis walk
- `wallet:create` (`createSchnorrInitializerlessAccount`): **69 ms** —
initializerless, no deploy tx, as
  designed
- `setup:bot` (4 nested hooks): **149,582 ms = 97.3%** of the file's
beforeAll
- transaction-bot `Bot.create`: 32.2s (token deploy class+instance, then
private+public mint)
  - bridge-resume `new BotStore(...)`: 0.9 ms
- amm-bot `AmmBot.create`: 73.9s (deploys 4 contracts + set_minter +
mint + add_liquidity)
- cross-chain-bot `CrossChainBot.create`: 43.5s (TestContract deploy +
seed 2 L1→L2 msgs + wait ready)

**Root cause: structural, not a bug.** The bot suite runs the production
Sequencer to exercise
CHECKPOINTED/PROPOSED follow modes and L1 fee-juice bridging, so each
`Bot.create` deploys real
contracts and mints serially at slot cadence (`bot/src/factory.ts`). The
two calls the shortlist
suspected are ~0.1s each.

No fix is shipped here because there is no safe one-liner: speeding this
up means batching/parallelizing
the `bot/src/factory.ts` deploy/mint paths (production code,
PR-6-shaped), genesis-seeding (PR 1/2), or
cutting the CI slot time (PR 5) — none belong in the instrumentation PR.
Tracked as a round-5 item; the
new `setup:bot` span makes that win measurable. Full write-up in
`tmp/SPEEDUP_ROUND4_FINDINGS.md`.

## Expected effect

No wall-clock change (passthrough spans). The deliverable is visibility:
the previously-invisible
`setup:bot` / `setup:wallet` / `wallet:create` decomposition of the bot
beforeAll, and the
`wait:proven-checkpoint` / `warp:proven-checkpoint-epoch` residual in
the fees suites.

Measured numbers from a full green CI run will be appended below once
available.

## Measured impact

Both runs are full CI runs. PR run: `ci/x-fast` on head `1f7898cd2b` (CI
1783345614459429, 1,799 test
lines). Baseline: `ci/x-full-no-test-cache` on the exact merge-base
`a8cfa93df5` (CI 1783341726388943,
1,863 test lines). No wall-clock change claimed (spans are passthrough;
suite-level noise floor is
~11s median / ~52s p90) — the deliverable is attribution, measured as
span `busyMs`.

Bot suite (`suite` line, beforeAll = 181.5s on the PR run vs 182.1s
baseline):

- Baseline: 7.4s tagged (`setup:env:*` only) → **174.8s / 96%
invisible**.
- PR run: `setup:bot` **174,203 ms** (4 occurrences, max 86,961 ms = the
amm-bot hook),
`setup:wallet` **250 ms**, `wallet:create` **144 ms**; residual now
**−1.5s ≈ 0** →
  **100% of the beforeAll is attributed**.
- This confirms the diagnosis: `EmbeddedWallet.create` +
`createSchnorrInitializerlessAccount` are
~0.4s combined; the "invisible 178s" is the four nested describes'
bot-factory setup
(deploys + mints at production 12s-slot cadence), folded into the file's
suite line by the
  hook accounting.

Fees suites (`private_payments.parallel` × 8 shards + `failures`):

- Baseline: avg **32.3s/shard untagged residual** (zero
`wait:proven-checkpoint` /
  `warp:proven-checkpoint-epoch` lines on these suites).
- PR run: `wait:proven-checkpoint` fires on all 9 suite hooks, **293,888
ms total**
(avg 32.7s/shard — matches the residual exactly), plus one in-test
occurrence (32.1s);
`warp:proven-checkpoint-epoch` 19 occurrences, 361 ms total (the warp
RPC itself is
near-free — the cost is entirely the proven-chain poll that follows).
Per-shard residual is now
  ≈ −1s, i.e. fully attributed.

Net: **~469s/run of previously invisible setup/wait cost is now tagged
and attributable**
(174.2s `setup:bot` + 0.4s wallet spans + 294.3s fees proven-chain
waits), with no behavior or
wall-clock change.

Fixes A-1407

(cherry picked from commit 1476179)
PhilWindle pushed a commit that referenced this pull request Jul 21, 2026
Parallelizes the independent setup steps in the bot factory
(`@aztec/bot`, `bot/src/factory.ts`).
This is production bot code that runs against live networks, so the
changes are strictly
behavior-preserving: same contracts at the same (salt-derived)
addresses, same amounts, same final
state. Only the ordering/concurrency of provably-independent setup steps
changes.

The bot factory is the single largest cost in the `single-node/bot` e2e
suite: each `*.create` runs
the production sequencer at 12s L2 slots (`PIPELINING_SETUP_OPTS`) and
deploys/mints one tx per slot,
fully serial. `AmmBot.create` alone was ~74s (7 serial txs).

## Dependency graph per bot type

Edges below are "must be mined before", verified against the Noir
contract source
(`token_contract`, `amm_contract`).

### Transaction bot -- `setup()` (unchanged)

`setupAccount` -> register recipient -> `ensureFeeJuiceBalance` ->
deploy token -> mint. This chain is
fully data-dependent: funding gates the deploy, the deploy gates the
mint. The recipient
(`createSchnorrAccount`) is register-only (no tx), and the
private+public mints are already batched
into one tx. Nothing is independent, so `setup()` is left serial.

### AMM bot -- `setupAmm()`

Steps: deploy token0 (D0), token1 (D1), LP token (DL), AMM (DA);
`set_minter` granting the AMM rights
over the LP token (SM); mint token0+token1 to the provider (M, one
batched tx); `add_liquidity` (AL).

- D0, D1, DL: independent -- distinct contracts, no cross-references.
- DA: the AMM constructor only stores the three token addresses (no
calls into the tokens). Those
addresses are salt-derived, so DA needs only the (pre-derived)
addresses, not the token deploys mined.
- SM: `Token::set_minter` just writes `minters[amm] = true`; it does not
call or validate the AMM
contract. It needs the LP token deployed and the (derivable) AMM address
only -- not the AMM deployed.
The deployer is authorized because the Token constructor sets
`minters[admin] = true`.
- M: `mint_to_private`/`mint_to_public` require the caller be a minter;
the deployer is admin=minter
from the constructor, so the token0/token1 mints need only those tokens
deployed (no `set_minter`).
- AL: `add_liquidity` transfers token0/token1 from the provider (needs M
for balances), mints LP
tokens (needs SM so the AMM is an LP minter), and targets the AMM (needs
DA).

Restructured from 7 serial txs into 3 slot-phases:

- Phase 1: `Promise.all([D0, D1, DL])`
- Phase 2: `Promise.all([DA, SM, M])`
- Phase 3: `AL`

### Cross-chain bot -- `setupCrossChain()`

Steps: deploy TestContract (an L2 tx), seed N L1->L2 messages (L1 txs),
wait for the first message ready.

- The seeds reference only the L2 recipient (TestContract) address,
which is salt-derived. L1->L2
messages are queued on L1 and do not require the L2 contract to exist
yet; they are consumed later in
`bot.run()`, after setup completes. So the L2 deploy and the L1 seeding
are independent.
- The L2 deploy pays via the L2 wallet/PXE account; the seeds use the
bot's L1 client -- different
  accounts, so there is no L1 nonce interaction between them.

Restructured to overlap the deploy with the seed loop:
`Promise.all([deploy TestContract, seed loop])`,
then the message-ready wait.

## What stayed serial, and why

- The whole transaction-bot `setup()` (data-dependent chain, nothing
independent).
- `ensureFeeJuiceBalance` before every deploy (funding must precede
spending).
- `add_liquidity` after its mints + minter grant + AMM deploy.
- The individual L1->L2 seeds inside the loop: they share the bot's
single L1 account, so concurrent
sends would race on the L1 nonce. The suite already documents this nonce
hazard. Only the loop as a
  whole overlaps the (L2) TestContract deploy.

## Production safety / idempotent re-entry

- Every deploy still goes through `registerOrDeployContract`, which
checks
`getContractMetadata(address).isContractPublished` per contract and only
registers (no tx) if already
deployed. If one parallel deploy fails and the others succeed, the next
`*.create` recovers: the
succeeded contracts register-only, the failed one redeploys. Addresses
are salt-derived and identical
  across runs.
- `set_minter` is idempotent (writes `true` again). The AMM path's mint
and `add_liquidity` always run
  (no balance guard) -- unchanged from the previous `fundAmm`.
- Same-sender concurrent `.send()`s are safe by construction: the PXE
serializes simulate/prove through
its `SerialQueue` and setup txs pay with random tx nonces. The bot runs
the same `EmbeddedWallet` ->
  PXE path in the e2e suite and in production.
- Deliberate, benign failure-path change: because Phase 2 runs DA/SM/M
concurrently, a failure of one no
longer prevents the others from being sent. Their effects (a
deployed-but-unused AMM, a minter grant,
an extra mint) are all idempotent-safe, and `add_liquidity` uses fixed
amounts, so re-entry converges
  to the same final state.
- `withNoMinTxsPerBlock` (the `flushSetupTransactions` save/zero/restore
wrapper around each setup tx)
was not safe to re-enter concurrently: interleaved save/zero/restore
could read the already-zeroed
value and "restore" `minTxsPerBlock` to 0 permanently. It now
reference-counts entrants -- the first
saves and zeroes, the last restores -- with unit tests covering the
overlapping and failure paths
(`bot/src/factory.test.ts`). Serial callers see the exact same RPC
sequence as before.

Final state is identical in every path: same contract addresses, same
minter grant, same minted
amounts, same liquidity.

## Local evidence

Two full local runs of `single-node/bot/bot.test.ts` (production
sequencer, 12s L2 slots), all 10
tests passing in both. Hook durations from factory log timestamps,
against the round-4 local baseline
measured on the same machine at the same cadence (findings from #24534):

- `Bot.create` (transaction-bot hook, untouched): 31.8s vs 32.2s
baseline -- unchanged, as intended.
- `AmmBot.create`: 57.2s vs 73.9s baseline (-16.7s). The three token
deploy txs go out within 300ms of
each other; the AMM deploy, `set_minter` and the mint batch all go out
within the following slot.
- `CrossChainBot.create`: 24.0s vs 43.5s baseline (-19.5s). The
TestContract deploy overlaps both L1
seeds; the first message is ready almost immediately after the deploy is
mined.
- Suite-level `beforeHooksMs`: 116.9s vs 153.7s baseline (-36.8s),
matching the per-hook deltas.
- Unit: `bot/src/factory.test.ts` (4 tests) covering the
`withNoMinTxsPerBlock` reentrancy fix.

## Expected effect on CI

- `AmmBot.create`: 7 serial txs -> 3 slot-phases (~87s on CI per
#24534's spans -> ~55-60s).
- `CrossChainBot.create`: deploy overlaps the seed pipeline (~43s ->
~25s).
- Transaction bot: unchanged.
- Aggregate: roughly -35 to -45s on the bot suite's beforeAll wall time.
- Fix-phase proof metric (once #24534's `setup:bot` instrumentation is
on this base): the amm-bot and
cross-chain `setup:bot` occurrences drop per the numbers above. Measured
CI numbers to be appended
  when a full run lands.

## Measured impact

The bot suite (a single shard) drops 216.3s → 118.3s total (**−98.0s,
−45%**); beforeHooks 215.9s →
118.0s.

- No `setup:bot` spans exist on this base (that instrumentation lives in
the unmerged #24534), so the
metric is the suite-level hook fold, which covers all three nested bot
factory setups (Bot, AmmBot,
  CrossChainBot).
- Noise context from the same run pair: untouched light suites move −2
to −3s per shard
(private_initialization −2.9s/shard over 20 shards, deploy_method
−2.1s/shard over 14), and the
largest counter-move is validators_sentinel +24s (multi-node variance).
A −98s single-shard delta is
  far outside that envelope.
- The CI delta exceeds the local −36.8s because the CI baseline pays
more missed-slot penalties per
serial tx (baseline beforeHooks 215.9s on CI vs 153.7s locally at the
same 12s cadence); collapsing
7 serial txs into 3 slot-phases removes proportionally more where slots
are more often missed.

Baseline CI run 1783393028773172 (merge-base 30966a4, full). PR CI run
1783427250198215 (x-fast).

Fixes A-1408

(cherry picked from commit 5d2b6df)
PhilWindle pushed a commit that referenced this pull request Jul 21, 2026
## What / why

Round-4 e2e speedup, PR 3 of the effort. The `single-node/bot` suite's
file-level `beforeAll` costs
~178s on CI with 97% untagged, and the residual proven-chain waits in
the `private_payments`/`failures`
fees suites (~32s/shard) were also invisible. This PR instruments both
so the cost is attributable, and
diagnoses the bot residual.

Spans are pure passthrough when `TEST_TIMING_FILE` is unset (see
`fixtures/timing.ts`), so this is zero
behavior change for normal runs and CI.

## Commit

- `test(e2e): instrument bot suite setup and fees proven-chain
residuals`
- Bot suite (`single-node/bot/bot.test.ts`): `setup:wallet` around
`EmbeddedWallet.create`,
`wallet:create` around `createSchnorrInitializerlessAccount`, and
`setup:bot` around the four nested
`describe` beforeAll hooks (`Bot.create` / `AmmBot.create` /
`CrossChainBot.create` / `BotStore`).
- Fees harness (`single-node/fees`): `wait:proven-checkpoint` around
`catchUpProvenChain`'s poll loop,
and `warp:proven-checkpoint-epoch` around `advanceToNextEpoch` via a new
instrumented
`FeesTest.advanceToNextEpoch()` wrapper; `waitForEpochProven` and the
direct call sites in
    `failures`/`private_payments` now route through it.
- Reuses existing tags where the concept matches (`wallet:create` is
already what
`createFundedInitializerlessAccounts` fires;
`warp:proven-checkpoint-epoch` is already what
`advanceToNextEpoch` fires in `block-production/setup.ts`) rather than
minting near-duplicates.

## Diagnosis

The timing environment (`shared/timing_env.mjs`) folds *every*
`beforeAll` in a file — the file-level
one plus each nested `describe`'s — into a single suite-scoped
`beforeHooksMs`, and every span fired
during any beforeAll lands on that same suite line. The shortlist read
the whole number as the
file-level hook and, seeing only `setup:env:*` tagged, attributed the
rest to the one visible untagged
file-level call (`createSchnorrInitializerlessAccount`). It is actually
the nested hooks.

Local read (`TEST_TIMING_SPANS=1`, `PIPELINING_SETUP_OPTS` = 12s L2
slots, production sequencer, fake
prover), suite `beforeHooksMs = 153,733 ms`:

- `setup:env:*` (file-level `setup()`): ~4.6s — already tagged
- `setup:wallet` (`EmbeddedWallet.create`, ephemeral): **137 ms** — the
PXE has `autoSync:false`, no
  genesis walk
- `wallet:create` (`createSchnorrInitializerlessAccount`): **69 ms** —
initializerless, no deploy tx, as
  designed
- `setup:bot` (4 nested hooks): **149,582 ms = 97.3%** of the file's
beforeAll
- transaction-bot `Bot.create`: 32.2s (token deploy class+instance, then
private+public mint)
  - bridge-resume `new BotStore(...)`: 0.9 ms
- amm-bot `AmmBot.create`: 73.9s (deploys 4 contracts + set_minter +
mint + add_liquidity)
- cross-chain-bot `CrossChainBot.create`: 43.5s (TestContract deploy +
seed 2 L1→L2 msgs + wait ready)

**Root cause: structural, not a bug.** The bot suite runs the production
Sequencer to exercise
CHECKPOINTED/PROPOSED follow modes and L1 fee-juice bridging, so each
`Bot.create` deploys real
contracts and mints serially at slot cadence (`bot/src/factory.ts`). The
two calls the shortlist
suspected are ~0.1s each.

No fix is shipped here because there is no safe one-liner: speeding this
up means batching/parallelizing
the `bot/src/factory.ts` deploy/mint paths (production code,
PR-6-shaped), genesis-seeding (PR 1/2), or
cutting the CI slot time (PR 5) — none belong in the instrumentation PR.
Tracked as a round-5 item; the
new `setup:bot` span makes that win measurable. Full write-up in
`tmp/SPEEDUP_ROUND4_FINDINGS.md`.

## Expected effect

No wall-clock change (passthrough spans). The deliverable is visibility:
the previously-invisible
`setup:bot` / `setup:wallet` / `wallet:create` decomposition of the bot
beforeAll, and the
`wait:proven-checkpoint` / `warp:proven-checkpoint-epoch` residual in
the fees suites.

Measured numbers from a full green CI run will be appended below once
available.

## Measured impact

Both runs are full CI runs. PR run: `ci/x-fast` on head `1f7898cd2b` (CI
1783345614459429, 1,799 test
lines). Baseline: `ci/x-full-no-test-cache` on the exact merge-base
`a8cfa93df5` (CI 1783341726388943,
1,863 test lines). No wall-clock change claimed (spans are passthrough;
suite-level noise floor is
~11s median / ~52s p90) — the deliverable is attribution, measured as
span `busyMs`.

Bot suite (`suite` line, beforeAll = 181.5s on the PR run vs 182.1s
baseline):

- Baseline: 7.4s tagged (`setup:env:*` only) → **174.8s / 96%
invisible**.
- PR run: `setup:bot` **174,203 ms** (4 occurrences, max 86,961 ms = the
amm-bot hook),
`setup:wallet` **250 ms**, `wallet:create` **144 ms**; residual now
**−1.5s ≈ 0** →
  **100% of the beforeAll is attributed**.
- This confirms the diagnosis: `EmbeddedWallet.create` +
`createSchnorrInitializerlessAccount` are
~0.4s combined; the "invisible 178s" is the four nested describes'
bot-factory setup
(deploys + mints at production 12s-slot cadence), folded into the file's
suite line by the
  hook accounting.

Fees suites (`private_payments.parallel` × 8 shards + `failures`):

- Baseline: avg **32.3s/shard untagged residual** (zero
`wait:proven-checkpoint` /
  `warp:proven-checkpoint-epoch` lines on these suites).
- PR run: `wait:proven-checkpoint` fires on all 9 suite hooks, **293,888
ms total**
(avg 32.7s/shard — matches the residual exactly), plus one in-test
occurrence (32.1s);
`warp:proven-checkpoint-epoch` 19 occurrences, 361 ms total (the warp
RPC itself is
near-free — the cost is entirely the proven-chain poll that follows).
Per-shard residual is now
  ≈ −1s, i.e. fully attributed.

Net: **~469s/run of previously invisible setup/wait cost is now tagged
and attributable**
(174.2s `setup:bot` + 0.4s wallet spans + 294.3s fees proven-chain
waits), with no behavior or
wall-clock change.

Fixes A-1407

(cherry picked from commit 1476179)
PhilWindle pushed a commit that referenced this pull request Jul 21, 2026
Parallelizes the independent setup steps in the bot factory
(`@aztec/bot`, `bot/src/factory.ts`).
This is production bot code that runs against live networks, so the
changes are strictly
behavior-preserving: same contracts at the same (salt-derived)
addresses, same amounts, same final
state. Only the ordering/concurrency of provably-independent setup steps
changes.

The bot factory is the single largest cost in the `single-node/bot` e2e
suite: each `*.create` runs
the production sequencer at 12s L2 slots (`PIPELINING_SETUP_OPTS`) and
deploys/mints one tx per slot,
fully serial. `AmmBot.create` alone was ~74s (7 serial txs).

## Dependency graph per bot type

Edges below are "must be mined before", verified against the Noir
contract source
(`token_contract`, `amm_contract`).

### Transaction bot -- `setup()` (unchanged)

`setupAccount` -> register recipient -> `ensureFeeJuiceBalance` ->
deploy token -> mint. This chain is
fully data-dependent: funding gates the deploy, the deploy gates the
mint. The recipient
(`createSchnorrAccount`) is register-only (no tx), and the
private+public mints are already batched
into one tx. Nothing is independent, so `setup()` is left serial.

### AMM bot -- `setupAmm()`

Steps: deploy token0 (D0), token1 (D1), LP token (DL), AMM (DA);
`set_minter` granting the AMM rights
over the LP token (SM); mint token0+token1 to the provider (M, one
batched tx); `add_liquidity` (AL).

- D0, D1, DL: independent -- distinct contracts, no cross-references.
- DA: the AMM constructor only stores the three token addresses (no
calls into the tokens). Those
addresses are salt-derived, so DA needs only the (pre-derived)
addresses, not the token deploys mined.
- SM: `Token::set_minter` just writes `minters[amm] = true`; it does not
call or validate the AMM
contract. It needs the LP token deployed and the (derivable) AMM address
only -- not the AMM deployed.
The deployer is authorized because the Token constructor sets
`minters[admin] = true`.
- M: `mint_to_private`/`mint_to_public` require the caller be a minter;
the deployer is admin=minter
from the constructor, so the token0/token1 mints need only those tokens
deployed (no `set_minter`).
- AL: `add_liquidity` transfers token0/token1 from the provider (needs M
for balances), mints LP
tokens (needs SM so the AMM is an LP minter), and targets the AMM (needs
DA).

Restructured from 7 serial txs into 3 slot-phases:

- Phase 1: `Promise.all([D0, D1, DL])`
- Phase 2: `Promise.all([DA, SM, M])`
- Phase 3: `AL`

### Cross-chain bot -- `setupCrossChain()`

Steps: deploy TestContract (an L2 tx), seed N L1->L2 messages (L1 txs),
wait for the first message ready.

- The seeds reference only the L2 recipient (TestContract) address,
which is salt-derived. L1->L2
messages are queued on L1 and do not require the L2 contract to exist
yet; they are consumed later in
`bot.run()`, after setup completes. So the L2 deploy and the L1 seeding
are independent.
- The L2 deploy pays via the L2 wallet/PXE account; the seeds use the
bot's L1 client -- different
  accounts, so there is no L1 nonce interaction between them.

Restructured to overlap the deploy with the seed loop:
`Promise.all([deploy TestContract, seed loop])`,
then the message-ready wait.

## What stayed serial, and why

- The whole transaction-bot `setup()` (data-dependent chain, nothing
independent).
- `ensureFeeJuiceBalance` before every deploy (funding must precede
spending).
- `add_liquidity` after its mints + minter grant + AMM deploy.
- The individual L1->L2 seeds inside the loop: they share the bot's
single L1 account, so concurrent
sends would race on the L1 nonce. The suite already documents this nonce
hazard. Only the loop as a
  whole overlaps the (L2) TestContract deploy.

## Production safety / idempotent re-entry

- Every deploy still goes through `registerOrDeployContract`, which
checks
`getContractMetadata(address).isContractPublished` per contract and only
registers (no tx) if already
deployed. If one parallel deploy fails and the others succeed, the next
`*.create` recovers: the
succeeded contracts register-only, the failed one redeploys. Addresses
are salt-derived and identical
  across runs.
- `set_minter` is idempotent (writes `true` again). The AMM path's mint
and `add_liquidity` always run
  (no balance guard) -- unchanged from the previous `fundAmm`.
- Same-sender concurrent `.send()`s are safe by construction: the PXE
serializes simulate/prove through
its `SerialQueue` and setup txs pay with random tx nonces. The bot runs
the same `EmbeddedWallet` ->
  PXE path in the e2e suite and in production.
- Deliberate, benign failure-path change: because Phase 2 runs DA/SM/M
concurrently, a failure of one no
longer prevents the others from being sent. Their effects (a
deployed-but-unused AMM, a minter grant,
an extra mint) are all idempotent-safe, and `add_liquidity` uses fixed
amounts, so re-entry converges
  to the same final state.
- `withNoMinTxsPerBlock` (the `flushSetupTransactions` save/zero/restore
wrapper around each setup tx)
was not safe to re-enter concurrently: interleaved save/zero/restore
could read the already-zeroed
value and "restore" `minTxsPerBlock` to 0 permanently. It now
reference-counts entrants -- the first
saves and zeroes, the last restores -- with unit tests covering the
overlapping and failure paths
(`bot/src/factory.test.ts`). Serial callers see the exact same RPC
sequence as before.

Final state is identical in every path: same contract addresses, same
minter grant, same minted
amounts, same liquidity.

## Local evidence

Two full local runs of `single-node/bot/bot.test.ts` (production
sequencer, 12s L2 slots), all 10
tests passing in both. Hook durations from factory log timestamps,
against the round-4 local baseline
measured on the same machine at the same cadence (findings from #24534):

- `Bot.create` (transaction-bot hook, untouched): 31.8s vs 32.2s
baseline -- unchanged, as intended.
- `AmmBot.create`: 57.2s vs 73.9s baseline (-16.7s). The three token
deploy txs go out within 300ms of
each other; the AMM deploy, `set_minter` and the mint batch all go out
within the following slot.
- `CrossChainBot.create`: 24.0s vs 43.5s baseline (-19.5s). The
TestContract deploy overlaps both L1
seeds; the first message is ready almost immediately after the deploy is
mined.
- Suite-level `beforeHooksMs`: 116.9s vs 153.7s baseline (-36.8s),
matching the per-hook deltas.
- Unit: `bot/src/factory.test.ts` (4 tests) covering the
`withNoMinTxsPerBlock` reentrancy fix.

## Expected effect on CI

- `AmmBot.create`: 7 serial txs -> 3 slot-phases (~87s on CI per
#24534's spans -> ~55-60s).
- `CrossChainBot.create`: deploy overlaps the seed pipeline (~43s ->
~25s).
- Transaction bot: unchanged.
- Aggregate: roughly -35 to -45s on the bot suite's beforeAll wall time.
- Fix-phase proof metric (once #24534's `setup:bot` instrumentation is
on this base): the amm-bot and
cross-chain `setup:bot` occurrences drop per the numbers above. Measured
CI numbers to be appended
  when a full run lands.

## Measured impact

The bot suite (a single shard) drops 216.3s → 118.3s total (**−98.0s,
−45%**); beforeHooks 215.9s →
118.0s.

- No `setup:bot` spans exist on this base (that instrumentation lives in
the unmerged #24534), so the
metric is the suite-level hook fold, which covers all three nested bot
factory setups (Bot, AmmBot,
  CrossChainBot).
- Noise context from the same run pair: untouched light suites move −2
to −3s per shard
(private_initialization −2.9s/shard over 20 shards, deploy_method
−2.1s/shard over 14), and the
largest counter-move is validators_sentinel +24s (multi-node variance).
A −98s single-shard delta is
  far outside that envelope.
- The CI delta exceeds the local −36.8s because the CI baseline pays
more missed-slot penalties per
serial tx (baseline beforeHooks 215.9s on CI vs 153.7s locally at the
same 12s cadence); collapsing
7 serial txs into 3 slot-phases removes proportionally more where slots
are more often missed.

Baseline CI run 1783393028773172 (merge-base 30966a4, full). PR CI run
1783427250198215 (x-fast).

Fixes A-1408

(cherry picked from commit 5d2b6df)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants